<!DOCTYPE html>
<html>
<head>
  <!-- hexo-inject:begin --><!-- hexo-inject:end --><meta charset="utf-8">
  
  <title>CRF Layer on the Top of BiLSTM - 8 | CreateMoMo</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
  <meta name="description" content="3.4 DemoIn this section, we will make two fake sentences which only have 2 words and 1 word respectively. Moreover, we will also randomly generate their true answers. Finally, we will show how to trai">
<meta property="og:type" content="article">
<meta property="og:title" content="CRF Layer on the Top of BiLSTM - 8">
<meta property="og:url" content="http://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/index.html">
<meta property="og:site_name" content="CreateMoMo">
<meta property="og:description" content="3.4 DemoIn this section, we will make two fake sentences which only have 2 words and 1 word respectively. Moreover, we will also randomly generate their true answers. Finally, we will show how to trai">
<meta property="og:locale" content="default">
<meta property="og:updated_time" content="2017-12-07T22:34:44.833Z">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="CRF Layer on the Top of BiLSTM - 8">
<meta name="twitter:description" content="3.4 DemoIn this section, we will make two fake sentences which only have 2 words and 1 word respectively. Moreover, we will also randomly generate their true answers. Finally, we will show how to trai">
  
  
    <link rel="icon" href="/favicon.png">
  
  
    <link href="//fonts.googleapis.com/css?family=Source+Code+Pro" rel="stylesheet" type="text/css">
  
  <link rel="stylesheet" href="/css/style.css"><!-- hexo-inject:begin --><!-- hexo-inject:end -->
  

</head>

<body>
  <!-- hexo-inject:begin --><!-- hexo-inject:end --><div id="container">
    <div id="wrap">
      <header id="header">
  <div id="banner"></div>
  <div id="header-outer" class="outer">
    <div id="header-title" class="inner">
      <h1 id="logo-wrap">
        <a href="/" id="logo">CreateMoMo</a>
      </h1>
      
    </div>
    <div id="header-inner" class="inner">
      <nav id="main-nav">
        <a id="main-nav-toggle" class="nav-icon"></a>
        
          <a class="main-nav-link" href="/">Home</a>
        
          <a class="main-nav-link" href="/archives">Archives</a>
        
      </nav>
      <nav id="sub-nav">
        
        <a id="nav-search-btn" class="nav-icon" title="Search"></a>
      </nav>
      <div id="search-form-wrap">
        <form action="//google.com/search" method="get" accept-charset="UTF-8" class="search-form"><input type="search" name="q" class="search-form-input" placeholder="Search"><button type="submit" class="search-form-submit">&#xF002;</button><input type="hidden" name="sitesearch" value="http://createmomo.github.io"></form>
      </div>
    </div>
  </div>
</header>
      <div class="outer">
        <section id="main"><article id="post-CRF-Layer-on-the-Top-of-BiLSTM-8" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/" class="article-date">
  <time datetime="2017-12-07T22:17:20.000Z" itemprop="datePublished">2017-12-07</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 class="article-title" itemprop="name">
      CRF Layer on the Top of BiLSTM - 8
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="3-4-Demo"><a href="#3-4-Demo" class="headerlink" title="3.4 Demo"></a>3.4 Demo</h4><p>In this section, we will make two fake sentences which only have 2 words and 1 word respectively. Moreover, we will also randomly generate their true answers. Finally, we will show how to train the CRF Layer by using Chainer v2.0. All the codes including the CRF layer are avaialbe from <a href="https://github.com/createmomo/CRF-Layer-on-the-Top-of-BiLSTM" target="_blank" rel="external">GitHub</a>.<br><a id="more"></a><br>Firstly, we import our own CRF Layer implmentation, ‘MyCRFLayer’.<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</div><div class="line"><span class="keyword">import</span> chainer</div><div class="line"></div><div class="line"><span class="keyword">import</span> MyCRFLayer</div></pre></td></tr></table></figure></p>
<p>We say that in our dataset we only have 2 labels (e.g. B-Person, O)<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">n_label = <span class="number">2</span></div></pre></td></tr></table></figure></p>
<p>The following code block is generating 2 sentences, xs = [x1, x2]. Sentence x1 has two words and x2 has only one word.<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">a = np.random.uniform(<span class="number">-1</span>, <span class="number">1</span>, n_label).astype(<span class="string">'f'</span>)</div><div class="line">b = np.random.uniform(<span class="number">-1</span>, <span class="number">1</span>, n_label).astype(<span class="string">'f'</span>)</div><div class="line"></div><div class="line">x1 = np.stack([b, a])</div><div class="line">x2 = np.stack([a])</div><div class="line"></div><div class="line">xs = [x1, x2]</div></pre></td></tr></table></figure></p>
<p>It should be noticed that, the elements of x1 and x2 are not the word embeddings instead of the emission scores from the BiLSTM layer which is not implemented here. </p>
<p>For example,in sentence x1 we have two words $w_0$ and $w_1$ and x1 is a matrix with shape of (2,2). The first “2” means it has two words and the second “2” denotes that in our dataset we have two labels as shown in the table below.</p>
<div class="table-container">
<table>
<thead>
<tr>
<th>x1</th>
<th>B-Person</th>
<th>O</th>
</tr>
</thead>
<tbody>
<tr>
<td>w0</td>
<td>0.1</td>
<td>0.65</td>
</tr>
<tr>
<td>w1</td>
<td>0.33</td>
<td>0.99</td>
</tr>
</tbody>
</table>
</div>
<p>Next we should have the true labels for these two sentences.<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">ys = [np.random.randint(n_label,size = x.shape[<span class="number">0</span>],dtype=<span class="string">'i'</span>) <span class="keyword">for</span> x <span class="keyword">in</span> xs]</div><div class="line"></div><div class="line">print(<span class="string">'Ground Truth:'</span>)</div><div class="line"><span class="keyword">for</span> i,y <span class="keyword">in</span> enumerate(ys):</div><div class="line">    print(<span class="string">'\tsentence &#123;0&#125;: [&#123;1&#125;]'</span>.format(str(i),<span class="string">' '</span>.join([str(label) <span class="keyword">for</span> label <span class="keyword">in</span> y])))</div></pre></td></tr></table></figure></p>
<p>Here are the ground truth generated randomly.</p>
<pre><code>Ground Truth:
    sentence 0: [0 0]
    sentence 1: [1]
</code></pre><p>Although we do not really have our BiLSTM layer, it will not affect us to show how to train a model in chainer. We simulated the outputs of BiLSTM layer and the true answers. Therefore, we can use some optimizers to optimize our CRF layer.</p>
<p>In this article, we used the Stochastic Gradient Descent method to train our model. (If now you are not familar with training methods, you can learn it in future.) This optimizer will update the parameters (i.e. the transition matrix) in our CRF Layer according to the loss between predicted labels and ground truth labels.<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">from</span> chainer <span class="keyword">import</span> optimizers</div><div class="line">optimizer = optimizers.SGD(lr=<span class="number">0.01</span>)</div><div class="line">optimizer.setup(my_crf)</div><div class="line">optimizer.add_hook(chainer.optimizer.GradientClipping(<span class="number">5.0</span>))</div></pre></td></tr></table></figure></p>
<p>The CRF layer is initialized by the number of labels (NOT including the extra added START and END).<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">my_crf = MyCRFLayer.My_CRF(n_label)</div></pre></td></tr></table></figure></p>
<p>Then we can start to train our CRF Layer.<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line">print(<span class="string">'Predictions:'</span>)</div><div class="line"></div><div class="line"><span class="keyword">for</span> epoch_i <span class="keyword">in</span> range(<span class="number">201</span>):</div><div class="line">    <span class="keyword">with</span> chainer.using_config(<span class="string">'train'</span>, <span class="keyword">True</span>):</div><div class="line">        loss = my_crf(xs,ys)</div><div class="line"></div><div class="line">        <span class="comment"># update parameters</span></div><div class="line">        optimizer.target.zerograds()</div><div class="line">        loss.backward()</div><div class="line">        optimizer.update()</div><div class="line"></div><div class="line">    <span class="keyword">with</span> chainer.using_config(<span class="string">'train'</span>, <span class="keyword">False</span>):</div><div class="line">        <span class="keyword">if</span> epoch_i % <span class="number">50</span> == <span class="number">0</span>:</div><div class="line">            print(<span class="string">'\tEpoch &#123;0&#125;: (loss=&#123;1&#125;)'</span>.format(str(epoch_i),str(loss.data)))</div><div class="line">            <span class="keyword">for</span> i, prediction <span class="keyword">in</span> enumerate(my_crf.argmax(xs)):</div><div class="line">                print(<span class="string">'\t\tsentence &#123;0&#125;: [&#123;1&#125;]'</span>.format(str(i), <span class="string">' '</span>.join([str(label) <span class="keyword">for</span> label <span class="keyword">in</span> prediction])))</div></pre></td></tr></table></figure></p>
<p>As shown in the outputs of our code, the loss is decreasing and the CRF layer is learning (the predictions are becoming correct).</p>
<pre><code>Predictions:
    Epoch 0: (loss=3.06651592255)
        sentence 0: [1 1]
        sentence 1: [1]
    Epoch 50: (loss=1.96822023392)
        sentence 0: [1 1]
        sentence 1: [1]
    Epoch 100: (loss=1.51349794865)
        sentence 0: [0 0]
        sentence 1: [1]
    Epoch 150: (loss=1.27118945122)
        sentence 0: [0 0]
        sentence 1: [1]
    Epoch 200: (loss=1.09977662563)
        sentence 0: [0 0]
        sentence 1: [1]
</code></pre><h4 id="3-5-GitHub"><a href="#3-5-GitHub" class="headerlink" title="3.5 GitHub"></a>3.5 GitHub</h4><p>The demo and CRF layer code are available on <a href="https://github.com/createmomo/CRF-Layer-on-the-Top-of-BiLSTM" target="_blank" rel="external">GitHub</a>. You will see, the code might not be perfect. Because for easy understanding, some implementations are very naive. I believe it could be optimized to a more efficient algorithm.</p>
<hr>
<p><strong>To end this series,</strong> I must say thank you to everyone who are reading these articles. Moreover, hope you enjoy the explanation of CRF layer.</p>
<p>Probably it is too early to say, anyway, <strong>HAPPY CHRISTMAS</strong> and <strong>HAPPY NEW YEAR</strong> :)!</p>
<hr>
<h2 id="References"><a href="#References" class="headerlink" title="References"></a>References</h2><p>[1]  Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K. and Dyer, C., 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.<br><a href="https://arxiv.org/abs/1603.01360" target="_blank" rel="external">https://arxiv.org/abs/1603.01360</a></p>
<blockquote>
<p>When you reprint or distribute this article, please include the original link address.</p>
</blockquote>

      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/" data-id="ck0lc6yzx0007ucp0ea68idze" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
    
<nav id="article-nav">
  
    <a href="/2018/01/17/My-Life/" id="article-nav-newer" class="article-nav-link-wrap">
      <strong class="article-nav-caption">Newer</strong>
      <div class="article-nav-title">
        
          My Life
        
      </div>
    </a>
  
  
    <a href="/2017/12/06/CRF-Layer-on-the-Top-of-BiLSTM-7/" id="article-nav-older" class="article-nav-link-wrap">
      <strong class="article-nav-caption">Older</strong>
      <div class="article-nav-title">CRF Layer on the Top of BiLSTM - 7</div>
    </a>
  
</nav>

  
</article>


<section id="comments">
  <div id="disqus_thread">
    <noscript>Please enable JavaScript to view the <a href="//disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
  </div>
</section>
</section>
        
          <aside id="sidebar">
  
    

  
    

  
    
  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Archives</h3>
    <div class="widget">
      <ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="/archives/2019/07/">July 2019</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2019/01/">January 2019</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2018/01/">January 2018</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/12/">December 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/11/">November 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/10/">October 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/09/">September 2017</a></li></ul>
    </div>
  </div>


  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Recent Posts</h3>
    <div class="widget">
      <ul>
        
          <li>
            <a href="/2019/07/18/Table-of-Contents/">Table of Contents</a>
          </li>
        
          <li>
            <a href="/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/">Probabilistic Graphical Models Revision Notes</a>
          </li>
        
          <li>
            <a href="/2018/01/23/Super-Machine-Learning-Revision-Notes/">Super Machine Learning Revision Notes</a>
          </li>
        
          <li>
            <a href="/2018/01/17/My-Life/">My Life</a>
          </li>
        
          <li>
            <a href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/">CRF Layer on the Top of BiLSTM - 8</a>
          </li>
        
      </ul>
    </div>
  </div>

  
</aside>
        
      </div>
      <footer id="footer">
  
  <div class="outer">
    <div id="footer-info" class="inner">
      &copy; 2019 CreateMoMo<br>
      Powered by <a href="http://hexo.io/" target="_blank">Hexo</a>
    </div>
  </div>
</footer>
    </div>
    <nav id="mobile-nav">
  
    <a href="/" class="mobile-nav-link">Home</a>
  
    <a href="/archives" class="mobile-nav-link">Archives</a>
  
</nav>
    
<script>
  var disqus_shortname = 'createmomo';
  
  var disqus_url = 'http://createmomo.github.io/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/';
  
  (function(){
    var dsq = document.createElement('script');
    dsq.type = 'text/javascript';
    dsq.async = true;
    dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
    (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>


<script src="//ajax.googleapis.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>


  <link rel="stylesheet" href="/fancybox/jquery.fancybox.css">
  <script src="/fancybox/jquery.fancybox.pack.js"></script>


<script src="/js/script.js"></script>

  </div>
<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
        tex2jax: {
            inlineMath: [ ["$","$"], ["\\(","\\)"] ],
            skipTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'],
            processEscapes: true
        }
    });
    MathJax.Hub.Queue(function() {
        var all = MathJax.Hub.getAllJax();
        for (var i = 0; i < all.length; ++i)
            all[i].SourceElement().parentNode.className += ' has-jax';
    });
</script>
<!-- <script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>-->
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_CHTML"></script><!-- hexo-inject:begin --><!-- hexo-inject:end -->
</body>
</html>